DATA ENGINEERING / EVENT-DRIVEN STREAMING

A raw weather API,
refined every 15 minutes.

AtmosFlow is an event-driven pipeline that streams live weather for 4 Vietnamese cities through Kafka into PostgreSQL, refines it with a dbt Bronze–Silver–Gold workflow on an Airflow schedule, and watches itself through Grafana.

View source code ↗

ROLE

Data Engineer

SCOPE

Individual project

DATA

Live weather API · 4 cities · 15-min cadence

STACK

Python · Kafka · PostgreSQL · dbt · Airflow · Grafana · Docker

01 / CONTEXT & PROBLEM

API responses aren't analytics. The distance between them is the pipeline.

A weather API returns point-in-time JSON — no history, no consistency guarantees, no aggregates. The exercise: build the full production pattern (streaming ingestion, layered modeling, scheduled orchestration, observability) at a scale one person can run and debug, so every component's failure mode is understood rather than abstracted away.

02 / ARCHITECTURE

Stream in, refine up

Weather API

4 Vietnamese cities

→

Python producer → Kafka

event stream · decoupled ingestion

→

PostgreSQL Bronze

raw landed events

→

dbt Silver → Gold

cleaned records → analytics-ready city summaries · full lineage

→

Grafana

pipeline & data monitoring

Airflow triggers the dbt refresh every 15 minutes · entire stack runs on Docker Compose

03 / KEY DECISIONS & RESULTS

Small data, production patterns

Kafka between API and database

Overkill for 4 cities, intentional for learning: the producer/consumer split means ingestion hiccups never block the warehouse, and replaying a window is a consumer restart.

Bronze–Silver–Gold with full lineage

Every Gold-layer city summary traces back through documented dbt models to the raw event that produced it — the same medallion discipline used on enterprise lakehouses.

Observability from day one

Grafana dashboards watch both the infrastructure and the data freshness — a pipeline that can't tell you it's broken isn't done.

15 min

automated end-to-end refresh cadence

3 layers

Bronze → Silver → Gold with documented lineage

4 cities

continuously streamed, modeled and monitored

04 / WHAT I'D DO DIFFERENTLY

Add contract tests at the API boundary.

The pipeline trusts the API's schema; a silent field change would flow into Bronze before anything complained. Next: schema validation at the producer with a quarantine topic, plus dbt source freshness checks wired into the Grafana alerts.